Benchmark Data and Evaluation Framework for Intent Discovery Around COVID-19 Vaccine Hesitancy

The COVID-19 pandemic has made a huge global impact and cost millions of lives. As COVID-19 vaccines were rolled out, they were quickly met with widespread hesitancy. To address the concerns of hesitant people, we launched VIRA, a public dialogue system aimed at addressing questions and concerns surrounding the COVID-19 vaccines. Here, we release VIRADialogs, a dataset of over 8k dialogues conducted by actual users with VIRA, providing a unique real-world conversational dataset. In light of rapid changes in users' intents, due to updates in guidelines or in response to new information, we highlight the important task of intent discovery in this use-case. We introduce a novel automatic evaluation framework for intent discovery, leveraging the existing intent classifier of VIRA. We use this framework to report baseline intent-discovery results over VIRADialogs, that highlight the difficulty of this task. © 2023 Association for Computational Linguistics.

MedConQA: Medical Conversational Question Answering System based on Knowledge Graphs

The medical conversational system can relieve doctors' burden and improve healthcare effi-ciency, especially during the COVID-19 pan-demic. However, the existing medical dialogue systems have die problems of weak scalability, insufficient knowledge, and poor controlla-bility. Thus, we propose a medical conversa-tional question-answering (CQA) system based on the knowledge graph, namely MedConQA, which is designed as a pipeline framework to maintain high flexibility. Our system utilizes automated medical procedures, including medi-cal triage, consultation, image-text drug recom-mendation, and record. Each module has been open-sourced as a tool, which can be used alone or in combination, with robust scalability. Besides, to conduct knowledge-grounded dia-logues with users, we first construct a Chinese Medical Knowledge Graph (CMKG) and col-lect a large-scale Chinese Medical CQA (CM-CQA) dataset, and we design a series of meth-ods for reasoning more intellectually. Finally, we use several state-of-the-art (SOTA) tech-niques to keep the final generated response more controllable, which is further assured by hospital and professional evaluations. We have open-sourced related code, datasets, web pages, and tools, hoping to advance future research. © 2022 Association for Computational Linguistics.

CDialog: A Multi-turn Covid-19 Conversation Dataset for Entity-Aware Dialog Generation

The development of conversational agents to interact with patients and deliver clinical advice has attracted the interest of many researchers, particularly in light of the COVID-19 pandemic. The training of an end-to-end neural based dialog system, on the other hand, is hampered by a lack of multi-turn medical dialog corpus. We make the very first attempt to release a high-quality multi-turn Medical Dialog dataset relating to Covid-19 disease named CDialog, with over 1K conversations collected from the online medical counselling websites. We annotate each utterance of the conversation with seven different categories of medical entities, including diseases, symptoms, medical tests, medical history, remedies, medications and other aspects as additional labels. Finally, we propose a novel neural medical dialog system based on the CDialog dataset to advance future research on developing automated medical dialog systems. We use pre-trained language models for dialogue generation, incorporating annotated medical entities, to generate a virtual doctor's response that addresses the patient's query. Experimental results show that the proposed dialog models perform comparably better when supplemented with entity information and hence can improve the response quality. © 2022 Association for Computational Linguistics.

A privacy-preserving dialogue system based on argumentation

Dialogue systems are a class of increasingly popular AI-based solutions to support timely and interactive communication with users in many domains. Due to the apparent possibility of users disclosing their sensitive data when interacting with such systems, ensuring that the systems follow the relevant laws, regulations, and ethical principles should be of primary concern. In this context, we discuss the main open points regarding these aspects and propose an approach grounded on a computational argumentation framework. Our approach ensures that user data are managed according to data minimization, purpose limitation, and integrity. Moreover, it is endowed with the capability of providing motivations for the system responses to offer transparency and explainability. We illustrate the architecture using as a case study a COVID-19 vaccine information system, discuss its theoretical properties, and evaluate it empirically. © 2022 The Author(s)

Where and Why is My Bot Failing? A Visual Analytics Approach for Investigating Failures in Chatbot Conversation Flows

The ongoing coronavirus pandemic has accelerated the adoption of AI-powered task-oriented chatbots by businesses and healthcare organizations. Despite advancements in chatbot platforms, implementing a successful and effective bot is still challenging and requires a lot of manual work. There is a strong need for tools to help conversation analysts quickly identify problem areas and, consequently, introduce changes to chatbot design. We present a visual analytics approach and tool for conversation analysts to identify and assess common patterns of failure in conversation flows. We focus on two key capabilities: path flow analysis and root cause analysis. Interim evaluation results from applying our tool in real-world customer production projects are presented.

Improve Chit-Chat and QA Sentence Classification in User Messages of Dialogue System using Dialogue Act Embedding

In recent years, dialogue system is booming and widely used in customer service system, and has achieved good results. Viewing the conversation records between users and real customer service, we can see that the user's sentences are mixed with questions about products and services, and chat with customer service. According to the experience of professionals, it is helpful in improving the user experience to mix some chats in customer service conversations. However, users' questions are expected to be answered, while chatting is expected to interact with customer service. In order to produce an appropriate response, the dialogue system must be able to distinguish these two intentions effectively. Dialog act is a classification that linguists define according to its function. We think this information will help distinguishing questioning sentences and chatting sentences. In this paper, we combine a published COVID-19 QA dataset and a COVID-19-topic chat dataset to form our experimental data. Based on the BERT (Bidirectional Encoder Representation from Transformers) model, we build a question-chat classifier model. The experimental results show that the accuracy of the configuration with dialog act embedding is 16% higher than that with only original statement embedding. In addition, it is found that conversation behavior types such as "Statement-non-opinion", "Signal-non-understanding" and "Appreciation" are more related to question sentences, while "Wh-Question", "Yes-No-Question" and "Rhetorical-Question" questions are more related to chat sentences. © 2021 ROCLING 2021 - Proceedings of the 33rd Conference on Computational Linguistics and Speech Processing. All rights reserved.

Speaker and time-aware joint contextual learning for dialogue-act classification in counselling conversations

The onset of the COVID-19 pandemic has brought the mental health of people under risk. Social counselling has gained remarkable significance in this environment. Unlike general goal-oriented dialogues, a conversation between a patient and a therapist is considerably implicit, though the objective of the conversation is quite apparent. In such a case, understanding the intent of the patient is imperative in providing effective counselling in therapy sessions, and the same applies to a dialogue system as well. In this work, we take forward a small but an important step in the development of an automated dialogue system for mental-health counselling. We develop a novel dataset, named HOPE, to provide a platform for the dialogue-act classification in counselling conversations. We identify the requirement of such conversation and propose twelve domain-specific dialogue-act (DAC) labels. We collect ∼ 12.9K utterances from publicly-available counselling session videos on YouTube, extract their transcripts, clean, and annotate them with DAC labels. Further, we propose SPARTA, a transformer-based architecture with a novel speaker- and time-aware contextual learning for the dialogue-act classification. Our evaluation shows convincing performance over several baselines, achieving state-of-the-art on HOPE. We also supplement our experiments with extensive empirical and qualitative analyses of SPARTA. © 2022 ACM.

On the Generation of Medical Dialogs for COVID-19

Under the pandemic of COVID-19, people experiencing COVID19-related symptoms have a pressing need to consult doctors. Because of the shortage of medical professionals, many people cannot receive online consultations timely. To address this problem, we aim to develop a medical dialog system that can provide COVID19-related consultations. We collected two dialog datasets - CovidDialog - (in English and Chinese respectively) containing conversations between doctors and patients about COVID-19. While the largest of their kind, these two datasets are still relatively small compared with generaldomain dialog datasets. Training complex dialog generation models on small datasets bears high risk of overfitting. To alleviate overfitting, we develop a multi-task learning approach, which regularizes the data-deficient dialog generation task with a masked token prediction task. Experiments on the CovidDialog datasets demonstrate the effectiveness of our approach. We perform both human evaluation and automatic evaluation of dialogs generated by our method. Results show that the generated responses are promising in being doctorlike, relevant to conversation history, clinically informative and correct. © 2021 Association for Computational Linguistics.

A Preliminary Evaluation of a Privacy-Preserving Dialogue System

Dialogue systems are AI applications widely used in many contexts requiring user interaction. However, unconstrained interaction may lead to users communicating sensitive data. This raises concerns about how these systems handle personal data, and about their compliance with relevant laws, regulations, and ethical principles. We propose to integrate advanced natural language processing techniques in a dialogue system architecture based on computational argumentation, ensuring that user data are ethically managed and regulations are respected. A preliminary experimental evaluation of our proposal over a COVID-19 vaccine information case study shows promising results. Copyright © 2021 Copyright for this paper by its authors.